Field of the Invention
The present invention relates to an image processing method of a parallax image.
Description of the Related Art
An image pickup apparatus such as a video camera and a digital camera requires having a thin shape and a high zoom ratio. In a conventional typical image pickup system, an optical system is configured by combining a plurality of optical lenses to suppress generation of an aberration of the optical system and also to satisfy a desired optical performance. In order to reduce in size of this optical system, reduction in image size and reduction in diameter of the optical system are considered. However, it is difficult to reduce the image size while the resolution is maintained.
On the other hand, a compound eye image pickup apparatus (compound eye image pickup apparatus) which divides an optical system into a plurality of optical systems to achieve small-size optical systems has been proposed. The term “compound eye” means a configuration using a structure of an eye of an insect, and for example a configuration in which the optical system is constituted by a lens array including a plurality of lens units and each lens unit is reduced in size and reduced in focal length to miniaturize the optical system is known. However, in the conventional compound eye image pickup apparatus, it is difficult to add an optical zoom function to make a shooting angle of view variable. This is because the optical zoom function which makes the shooting angle of view variable by using a method of moving a position of a lens constituting the optical system needs a mechanical moving mechanism and accordingly the image pickup system is increased in size.
Japanese Patent Laid-open No. 2005-303694 discloses a configuration in which a lens unit with a short focus and a lens unit with a long focus which have angles of view different from each other are disposed to capture an image to include the same part of an object. In other words, a zoom-up image obtained by an image pickup element corresponding to the long focus lens is fitted into part of a wide image obtained by an image pickup element corresponding to the short focus lens. Accordingly, an image which has a high resolution for part of an area of the image and has a low resolution and a wide angle of view for the other area of the image can be obtained. This is effective for a surveillance camera, and for example a suspicious figure at a center area or the like can be zoomed up to be monitored in detail while understanding an entire area of a monitoring area.
Japanese Patent Laid-open No. 2011-135359 discloses a compound eye image pickup system which is configured by a plurality of sub-camera modules in which color filters different from each other are disposed. In addition, Japanese Patent Laid-open No. 2011-135359 discloses a method of performing restoration processing on an image for each color component to correct an aberration of an optical system which easily occurs as the image pickup system is thinned.
The compound eye image pickup apparatus is capable of acquiring the “Light Field” by disposing image pickup systems in an array to obtain parallax images. The “Light Field” means information of a position and an angle of a ray from an object taken into the image pickup apparatus. In detail, the “Light Field” is described in “Ren.Ng, etc., ‘Light Field Photography with a Hand-held Plenoptic Camera’, Stanford Tech Report CTSR 2005-2”. A plurality of configurations of image pickup apparatus which are capable of acquiring the “Light Field” are known, and the compound eye image pickup apparatus is one of them. The “Light Field” obtained as an image on an image pickup element is a parallax image or an image corresponding to a pupil of an optical system according to the configuration of the image pickup apparatus, and it is the same in that the ray passing through the pupil of the optical system is separated into the position and the angle to be obtained. That is, the parallax image and the image corresponding to the pupil of the optical system can be treated to be approximately equivalent by rearranging information of each pixel. When the “Light Field” is acquired and the image is reconstructed by image processing, focusing and depth adjustment within a predetermined range can be performed after shooting (i.e., capturing) the image. Such a function is beneficial to a thin compound eye image pickup apparatus to reduce drive units.
However, in the configuration of Japanese Patent Laid-open No. 2005-303694, optical axes of the optical systems with focal lengths different from each other are displaced from each other. Therefore, it is not appropriate for performing continuous zooming in which viewpoints are constant from a wide-angle side to a telephoto side similarly to a typical monocular zoom optical system. Furthermore, in this configuration, “Light Field” cannot be acquired.
In an actual shooting lens, not a little aberration exists. A spread of light beams at a point which is conjugate to one point in an object space is called a point spread function (PSF). A color bleeding, such as an axial chromatic aberration, a spherical aberration of a color, and a coma aberration of the color, in a color image, and a color shift in a lateral direction, that is, a chromatic aberration of magnification can correspond to a difference in position or shape of the PSF for each wavelength. When an aberration occurs in each optical system in the compound eye image pickup apparatus, a reconstructed image is deteriorated because an image formed via each optical system is deteriorated. The method disclosed in Japanese Patent Laid-open No. 2011-135359 performs the restoration processing on an image formed via each of the optical systems in which color filters different from each other are disposed. However, since a plurality of optical systems with the same focal length are disposed, zooming cannot be performed. Furthermore, the color filters are different from each other although the plurality of optical systems with the same focal length are disposed, and accordingly the “Light Field” cannot be acquired.